Redis Pipeline for Performance Gains

Redis pipeline overview#

Redis follows a simple request-response pattern where clients, such as applications and Redis CLI, send requests to the Redis server over TCP and wait (block) for the response. Because Redis is a high performance database, this works well in most cases, even at a high volume of requests per second. However, another factor is the network latency between Redis (server) and the client(s). This is independent of Redis performance. Even if Redis is operating at peak performance, we’ll need to incur the cost of back and forth between the client and server for every request, which is called round-trip time.

Redis has a feature called pipeline that can help avoid this and thereby improve performance. By using a pipeline-based connection, we can send multiple commands without waiting for the response of an individual command. You can think of a pipeline as an optimization technique that relies on batching and sending many commands in a single request and collecting all the responses together.

Let’s see a pipeline in action with a demo application.

Application overview#

This example will sets the value of multiple items using a for loop. We’ll first try this by using the normal way (without a pipeline) and then we’ll apply pipelining. Finally, we’ll compare the time taken in both the cases. This will give us a good understanding of the performance benefits of pipelining.

First, let’s walk through the code.

Code explanation#

In the no_pipeline function, we follow the traditional way of executing operations:

In the code above we:

• Line 3: Note the time when the process started.

• Lines 6–7: Start a for loop and set the key/value one by one.

• Line 11: Finally, we print the time taken by the function.

Now, let’s look at the interesting pipelining approach in the Pipeline function:

In the code above, we:

• Line 3: Just like before, we note the time when the process started

• Line 10: We also start a for loop to set the key-value one by one.

• Line 5: Instead of using the *redis.Client, were using a *redis.Pipeline, which was obtained using the Pipeline function.

• Line 13: Just calling pipe.Set is not going to suffice though. We need to execute the pipeline; we do so by calling Exec.

• Line 18: Finally, we print the time taken by the function.

Testing the application#

Press “Click to Connect...” to bootstrap the application. Then, follow the instructions below to execute commands to test the application in the terminal:

First, invoke the application in default mode (without using a pipeline) with 10 items, and note the completion time:

You’ll get an output similar to below. Please note that the exact time will differ in your case.

Now, let’s invoke the application in pipeline mode with the same number of items (10) and note the completion time:

You’ll get an output similar to below. Please note that the exact time will differ in your case.

Notice the difference in the time taken. The pipeline mode works significantly faster because the commands are queued and executed all at once.

To verify using Redis CLI as well:

We can experiment with a different number of items to see how pipelining works for a higher workload.

Let’s invoke the application in default mode with 1000 items and note the completion time:

You’ll get an output similar to below. Please note that the exact time will differ in your case.

Now, let’s invoke the application in pipeline mode with the same number of items (1000) and note the completion time:

You’ll get an output similar to below. Please note that the exact time will differ in your case.

Just like before, notice the difference in the time taken. The pipeline mode works significantly faster (around 60 times faster in this case) because the commands are queued and executed all at once.

To verify using Redis CLI:

Let’s increase the number of items to a significantly high number—1 million items (1,000,000)—and observe the output.

Invoke the application in default mode. Please wait for the operation, it might take over a minute to complete, although the exact time might differ in your case.

You’ll get an output similar to below:

Now, let’s invoke the application in pipeline mode:

You’ll get an output similar to below. Please note that the exact time will differ in your case.

As the number of items increase, the performance gap between pipeline mode and default mode increases. Therefore, the greater the number of items being processed, the more likely we are to get more benefits by using pipelining.

To verify using Redis CLI:

Continue to try out the solution with different numbers. One thing that will remain the same in all the cases is the fact that the pipeline mode of execution will be better in terms of time when compared to the default (sequential) mode.

Caveats when using pipeline#

Pipelining is a great technique, but it’s important to be mindful about when it can be used.

• Memory usage: While we send commands using pipelining, Redis queues the replies, which uses RAM (memory). We should be mindful about this and try to batch commands when working with a large number of items in a pipeline operation (like the example above, which has 1 million items).

• Read after write semantics: As evident by now, a pipelined operation batches and returns the responses together. Therefore, it’s not possible to read the result of a write operation in the same pipeline, for example, if we need to get the value of a key that we have set as part of the pipeline.

• Pipelines in Redis Cluster: In a Redis Cluster, we need to ensure that the pipeline is executed on a set of keys that are on the same hash slot. Otherwise, Redis will respond with an error stating that the keys we’re trying to operate on are keys that belong to different hash slots.

Summary#

In this lesson, we learned about the Redis pipeline and explored how it can provide performance improvements with the help of practical examples.